Image encoder, image decoder, image encoding method, and image decoding method

ABSTRACT

An input image signal  101  is divided into MC block units and, when coding processing is performed in these divided units, a motion compensation section  107  generates a motion-compensated prediction image  106   a  by detecting movement amounts in predetermined MC block units, a smoothing filter section  124  performs, with respect to the prediction image  106   a , smoothing of pixels located at the boundaries of adjoining MC blocks on the basis of predetermined evaluation criteria, and a prediction residual signal  108 , which is obtained from the difference between the prediction image  106   b  obtained by the smoothing, and the input image (input image signal  101 ), is encoded. Accordingly, it is possible to use relatively straightforward computation to perform processing, with respect to a prediction frame image generated by block-unit motion-compensated interframe prediction (MC), to adaptively smooth a discontinuous waveform generated between MC blocks of the prediction frame image, whereby the efficiency of low bit rate coding that employs interframe MC is improved.

TECHNICAL FIELD

[0001] The present invention relates to an image coding apparatus, animage decoding apparatus, an image coding method, and an image decodingmethod that perform the transmission and storage of images with a smallencoding data volume and that are applied to a mobile image transmissionsystem or similar.

BACKGROUND ART

[0002] Conventional image coding systems are constituted to divide imageframes into blocks of a fixed size and then perform coding processing inthese divided units. Typical examples of conventional image codingsystems include the MPEG (Moving Picture Experts Group) 1 coding systemas described in Le Gall. D: “MPEG: A Video Compression Standard forMultimedia Applications”, Trans. ACM, 1991, April.

[0003] MPEG 1 performs motion-compensated interframe prediction (MC:Moving Compensation) by dividing image frames into fixed block unitsknown as macroblocks, detecting movement amounts (or motion vectors) byreferencing a local decoding frame image encoded in units, specifyingsimilar blocks from within a reference image and employing these similarblocks as predictive data. By means of this technique, even when thereis motion in an image, the prediction efficiency can be improved bytracking the motion, and redundancy in a temporal direction can bereduced. Furthermore, redundancy that remains in a spatial direction canbe reduced by employing the DCT (Discrete Cosine Transform), using unitsthat are blocks consisting of 8×8 pixels, with respect to a predictionresidual signal. A variety of standard image coding systems that startwith MPEG1 perform data compression of image signals by combining MC andthe DCT.

[0004]FIG. 20 is a block diagram showing the constitution of aconventional image coding apparatus based on an MPEG1 image codingsystem. An input image signal 1 which is inputted to the image codingapparatus shown in FIG. 20 is a temporal array of frame images and willsubsequently embody the signal of each frame image unit. Further, anexample of a frame image that is to be encoded is shown in FIG. 21. Thecurrent frame 601 is divided into fixed square/rectangular regions of 16pixels×16 lines (called macroblocks), and the processing that follows isperformed in these units.

[0005] The macroblock data of the current frame (current macroblocks)produced by the input image signal 1 are first outputted to a motiondetection section 2 where detection of motion vectors 5 is carried out.A motion vector 5 is detected by referencing a predetermined searchregion of a previous encoded frame image 4 (called a local decodingimage 4 hereinafter) stored in a frame memory 3, locating a patternsimilar to the current macroblock (called a prediction image 6hereinafter), and determining the amount of spatial displacement betweenthis pattern and the current macroblock.

[0006] Here, the local decoding image 4 is not limited to only theprevious frame. Rather, a future frame can also be used as a result ofbeing encoded in advance and stored in the frame memory 3. Although theuse of a future frame generates switching of the coding order and inturn an increase in the processing delay, there is the merit thatvariations in the image content produced between previous and futureframes is easily predicted, thus making it possible to effectivelyreduce temporal redundancy still further.

[0007] Generally, in MPEG1, it is possible to selectively use threecoding types which are called bidirectional prediction (B frameprediction), forward prediction (P frame prediction) that uses previousframes alone, and I frame prediction which does not perform interframeprediction, instead performing coding only within the frame. FIG. 21 islimited to P frame prediction alone, a local decoding image 4 beingrecorded with a previous frame 602.

[0008] The motion vector 5 shown in FIG. 20 is rendered by atwo-dimensional parallel displacement amount. Block matching asrepresented in FIGS. 22A to 22D is generally used as the method ofdetecting the motion vector 5. A motion search range 603 centered on thespatial phase of the current macroblock is established, then, from theimage data 604 within the motion search range 603 of the previous frame602, the block for which the sum of the squares of the differences orthe sum of the absolute values of differences is minimum is determinedas the motion predictive data, and the relocation amount of the motionpredictive data in relation to the current macroblock is determined asthe motion vector 5.

[0009] The motion predictive data for all the macroblocks in the currentframe is determined, and this data, which is rendered as a frame image,is equivalent to the motion prediction frame 605 in FIG. 21. Thedifference 606 between the motion prediction frame 605 shown in FIG. 21obtained by way of the above MC processing, and the current frame 601,is obtained (obtained by the subtraction section 21 shown in FIG. 20),and this residual signal (called the prediction residual signal 8hereinafter) undergoes DCT coding. Specifically, the processing toextract the motion predictive data for every macroblock (predictionimage 6 hereinafter) is performed by a motion compensation section 7.The processing performed by the motion compensation section 7 involvesusing a motion vector 5 to extract the prediction image 6 from the localdecoding image 4 stored in the frame memory 3.

[0010] The prediction residual signal 8 is converted into DCTcoefficient data 10 (also called DCT coefficients 10 hereinafter) by aDCT section 9. As shown in FIG. 23, the DCT converts spatial pixelvectors denoted by 610 into a set of normalized orthogonal bases thatrender fixed frequency components denoted by 611. 8×8 pixel blocks (‘DCTblocks’ below) are normally adopted for the spatial pixel vectors.Because the DCT is discrete transform processing, the DCT actuallyperforms conversion for each of the horizontal and vertical 8dimensional row and column vectors of the DCT block.

[0011] The DCT uses the correlation between pixels present in a spatialregion to localize the power concentration in the DCT block. The higherthe power concentration, the better the conversion efficiency is, andtherefore the performance of the DCT with respect to a natural imagesignal is not inferior when compared with a KL transform which is theoptimum transform. Particularly in the case of a natural image, thepower is concentrated in the lower regions including the DC component asa main part, and there is barely any power in the higher regions, andtherefore, as shown in FIG. 24, by scanning from the lower regions tothe higher regions as indicated by the arrows in the DCT block such thatthe quantized coefficients denoted by 612 are denoted by 613, and byincluding a large zero run, the overall coding efficiency which alsoincludes the results of entropy coding is improved.

[0012] The quantization of the DCT coefficients 10 is performed by aquantization section 11 and the quantized coefficients 12 obtainedthereby are scanned, run-length encoded, and multiplexed in a compressedstream 14 by a variable length coding section 13 before beingtransmitted. Further, the motion vectors 5 detected by the motiondetection section 2 are also multiplexed in the compressed stream 14 andtransmitted, one macroblock at a time, because these vectors arerequired in order to allow the image decoding apparatus describedsubsequently to generate a prediction image that is the same as that ofthe image coding apparatus.

[0013] In addition, the quantized coefficients 12 are decoded locallyvia a reverse quantization section 15 and a reverse DCT section 16, andthe decoded results are added to the prediction image 6 by an additionsection 22, whereby a decoding image 17 which is the same as that of theimage decoding apparatus is generated. The decoding image 17 is used inthe prediction for the next frame and is therefore stored in the framememory 3.

[0014] A description is provided next for the constitution of aconventional image decoding apparatus that is based on an MPEG1 imagedecoding system as shown in FIG. 25. After receiving the compressedstream 14, the image decoding apparatus detects a sync word indicatingthe start of each frame by means of a variable length decoding section18, and subsequently decodes motion vectors 5 and quantized DCTcoefficients 12 in macroblock units. The motion vectors 5 are outputtedto a motion compensation section 7 d and the motion compensation section7 d extracts, as a prediction image 6, the image parts which have movedto an extent equivalent to the motion vectors 5, from a frame memory 19(used in the same way as the frame memory 3), this operation beingsimilar to the operation of the above-mentioned image coding apparatus.The quantized DCT coefficients 12 are decoded via a reverse quantizationsection 15 d and a reverse DCT section 16 d, and then added by theaddition section 23 to the prediction image 6 to form the final decodingimage 17. The decoding image 17 is outputted using predetermined displaytiming to a display device (not shown) where the image is played back.

DISCLOSURE OF THE INVENTION

[0015] However, in a conventional apparatus, MC performs movement amountdetection based on the premise that all of the pixels in the blocks(referred to as MC blocks hereinafter and as macroblocks in the MPEG1example above) which are the units of MC possess the same motion.Consequently, the possibility exists that, with a prediction image thatis constituted by the spatial disposition of MC blocks, a signalwaveform will arise in which discontinuity is perceived at theboundaries of the MC blocks. This discontinuous waveform can becompensated by supplementing the residual component in cases where anadequate encoding data amount is allocated to the residual signal.However, when coding is carried out with a high compression ratio, asatisfactory rendition of the residual signal is not possible and thediscontinuous boundaries are sometimes apparent and perceived asdistortion.

[0016] Further, it has been identified that, because the DCT is also aclosed orthogonal transform in fixed blocks, in cases where thetransform basis coefficients are reduced as a result of coarsequantization, the signal waveform which naturally connects betweenblocks cannot be reconstituted and unnatural distortion is generatedbetween blocks (block distortion).

[0017] As means for solving the former MC block boundary discontinuity,overlapped motion compensation (called OBMC hereinafter) has beenproposed. As illustrated by FIGS. 26A and 26B, OBMC is a techniqueaccording to which predictive data specified by the MC blocks' ownmotion vectors is added weighted with predictive data specified bymotion vectors possessed by neighboring MC blocks, whereby the finalpredictive data is obtained.

[0018] In FIG. 26A, frame F(t) uses frame F(t−1) as a reference imageand extracts predictive data in units of MC blocks (A to E, for example)from within the reference image. Normal MC employs this predictive dataas is but OBMC extracts predictive data that corresponds to the positionof block C by using the motion vectors MV(A), MV(B), MV(C), and MV(D) ofthe neighboring blocks A, B, D, and E shown in FIG. 26B, in determiningthe prediction image Pc of block C. In this extraction, P{C,MV(A)}signifies processing to extract predictive data for the position of C byusing MV(A) The extracted predictive data are added weighted by W1 to W5as per the following formula.

Pc=W 1×P{C, MV(C)}+W 2×P{C, MV(A)}+W 3×P{C, MV(B)}+W 4×P{C, MV(D)}+W5×P{C, MV(E)}

[0019] Here, the weight is normally set so that the influence of theoriginal predictive data of block C becomes gradually smaller whenmoving from the center of the block C toward the block boundaries. Suchprocessing affords the benefit that, because the prediction image isdetermined such that the movement amounts of neighboring regions overlapblock C's own motion, the continuity of the waveform is preservedbetween the inner and outer pixels of the MC blocks and thus theboundaries thereof do not readily stand out.

[0020] However, with OBMC, there is the problem that, in addition to theextraction of predictive data for a block's own motion vectors,processing is also executed for all the MC blocks which includesprocessing to extract predictive data by means of motion vectors ofneighboring MC blocks, and processing that involves the weightedaddition of such data, meaning that the computational load is high.

[0021] Furthermore, in the movement amount detection involved in imagecoding, because detection is performed based on the criterion that thepower of the prediction residual should be minimized beyond the movementamount that matches the natural movement of the subject, the problemsexist that motion which is not based on true movement is sometimesdetected in regions containing a lot of noise or in other locations, andthat, in such locations, MC blocks are smoothed beyond what is necessarythrough the combined influence of neighboring movement amounts in OBMC,and two-line blurring is generated, and so forth.

[0022] On the other hand, as means for solving the latter DCT blockdistortion, a loop filter has been proposed. The loop filter acts as asmoothing filter for the boundaries of the DCT blocks of a decodingimage that is obtained by adding a prediction residual signal, which hasundergone encoding and local decoding, to a prediction image. This is atechnique that does not introduce the effects of distortion, as causedby DCT quantization, to MC by removing block distortion from thereference image which is used for subsequent frames. However, so long asMC is limited to being performed in block units, the discontinuitybetween MC blocks will not necessarily be avoided. Further, there is theproblem that in cases where residual coding which is not dependent onblock structure such as subband coding or a block-spanning basistransform, or the like, is performed, coding efficiency disadvantagescaused by the existence of a discontinuous waveform at block boundariescannot be avoided.

[0023] The present invention was conceived in view of these problems, anobject thereof being to provide an image coding apparatus, imagedecoding apparatus, image coding method, and image decoding method thatmake it possible to use relatively simple computation to performprocessing, with respect to a prediction frame image generated byblock-based motion-compensated interframe prediction (MC), to adaptivelysmooth a discontinuous waveform generated between MC blocks of theprediction frame image, whereby the efficiency of low bit rate codingthat employs interframe MC can be improved.

[0024] In order to resolve the above problems, the image codingapparatus according to the present invention is characterized bycomprising: motion compensation predicting means for generating amotion-compensated prediction image by detecting movement amounts inpredetermined partial image region units of an input image; smoothingmeans for performing smoothing of pixels located at the boundaries ofadjoining partial image regions on the basis of predetermined evaluationcriteria, with respect to the prediction image obtained by the motioncompensation predicting means; and prediction residual coding means forcoding a prediction residual signal obtained from the difference betweenthe input image and the smoothed prediction image.

[0025] Further, as the image decoding apparatus which corresponds tothis image coding apparatus, the image decoding apparatus according tothe present invention is characterized by comprising: motioncompensation predicting means for generating a motion-compensatedprediction image by detecting movement amounts in predetermined partialimage region units; smoothing means for performing smoothing of pixelslocated at the boundaries of adjoining partial image regions on thebasis of predetermined evaluation criteria, with respect to theprediction image obtained by the motion compensation predicting means;prediction residual decoding means for decoding a prediction residualsignal from the encoding side; and adding means for obtaining a decodedimage by adding together a decoded prediction residual signal obtainedby the prediction residual decoding means, and the smoothed predictionimage.

[0026] In addition, in order to resolve the above problems, the imagecoding method according to the present invention is characterized bycomprising: a motion compensation predicting step of generating amotion-compensated prediction image by detecting movement amounts inpredetermined partial image region units of an input image; a smoothingstep of performing smoothing of pixels located at the boundaries ofadjoining partial image regions on the basis of predetermined evaluationcriteria, with respect to the prediction image obtained by the motioncompensation predicting step; and a prediction residual coding step ofcoding the prediction residual signal obtained from the differencebetween the input image and the smoothed prediction image.

[0027] Further, as the image decoding method which corresponds to thisimage coding method, the image decoding method according to the presentinvention is characterized by comprising: a motion compensationpredicting step of generating a motion-compensated prediction image bydetecting movement amounts in predetermined partial image region units;a smoothing step of performing smoothing of pixels located at theboundaries of adjoining partial image regions on the basis ofpredetermined evaluation criteria, with respect to the prediction imageobtained by the motion compensation predicting step; a predictionresidual decoding step of decoding a prediction residual signal from theencoding side; and an adding step of obtaining a decoded image by addingtogether a decoded prediction residual signal obtained by the predictionresidual decoding step, and the smoothed prediction image.

[0028] According to this constitution, smoothing is performed for pixelslocated at the boundaries of adjoining partial image regions on thebasis of predetermined evaluation criteria, with respect to theprediction image, and it is therefore possible to perform correction inthe direction in which only the smoothing processing, that correctsdiscontinuity in partial image regions, is allowed. It is thereforepossible to improve the coding efficiency by suppressing discontinuouswaveforms generated in the prediction residual. Accordingly, it ispossible to use relatively straightforward computation to performprocessing, with respect to a prediction frame image generated byblock-unit motion-compensated interframe prediction (MC), to adaptivelysmooth a discontinuous waveform generated between MC blocks of theprediction frame image, whereby the efficiency of low bit rate codingthat employs interframe MC can be improved.

BRIEF DESCRIPTION OF THE DRAWINGS

[0029]FIG. 1 is a block diagram showing the constitution of the imagecoding apparatus according to a first embodiment of the presentinvention;

[0030]FIG. 2 is a block diagram showing the constitution of the imagedecoding apparatus according to the first embodiment;

[0031]FIGS. 3A to 3G show the form of MC (motion-compensated interframeprediction) blocks;

[0032]FIG. 4 serves to illustrate the problems posed with block unit MC;

[0033]FIG. 5 is a block diagram showing the constitution of a smoothingfilter section of the first embodiment;

[0034]FIG. 6 is a flowchart serving to illustrate the computationalprocessing operation of the block activity level calculation section ofthe smoothing filter section;

[0035]FIG. 7 shows an example of a unit for determining the blockactivity level;

[0036]FIG. 8 is a flowchart to illustrate the operation of theprocessing to correct futile filter processing in the smoothingfiltering performed by the filter processing section of the smoothingfilter section;

[0037]FIG. 9 shows the appearance of pixels for processing which arebetween blocks that are adjoined in a lateral direction, in smoothingfilter processing;

[0038]FIGS. 10A and 10B show the appearance of pixels for processingwhich are between blocks that are adjoined in a lateral direction, insmoothing filter processing performed by another filter;

[0039]FIG. 11 shows the function established in the post-processingsection of the smoothing filter section;

[0040]FIG. 12 is a block diagram showing the constitution of the imagecoding apparatus according to a second embodiment of the presentinvention;

[0041]FIG. 13 is a block diagram showing the constitution of the imagedecoding apparatus according to the second embodiment;

[0042]FIG. 14 serves to illustrate the definition, for block boundaries,of an activity level, in the smoothing filter section according to athird embodiment of the present invention;

[0043]FIGS. 15A and 15B serve to illustrate a case where a 5-tap filter,which has the pixel to be filtered at the center and which uses twopixels to the left and right thereof respectively, is employed in thesmoothing filter section according to a fourth embodiment of the presentinvention;

[0044]FIG. 16 is a block diagram showing the constitution of the imagecoding apparatus according to a fifth embodiment of the presentinvention;

[0045]FIG. 17 is a block diagram showing the constitution of the imagedecoding apparatus according to the fifth embodiment;

[0046]FIG. 18 is a block diagram showing the constitution of thesmoothing filter section according to the fifth embodiment;

[0047]FIG. 19 is a flowchart to illustrate the operation of the blockactivity level calculation section in the smoothing filter section ofthe fifth embodiment;

[0048]FIG. 20 is a block diagram showing the constitution of aconventional image coding apparatus based on an MPEG1 image codingsystem;

[0049]FIG. 21 is a conceptual view of motion-compensated frameprediction;

[0050]FIGS. 22A to 22D are conceptual views of motion compensation bymeans of block matching;

[0051]FIG. 23 is a conceptual view of the Discrete Cosine Transform;

[0052]FIG. 24 is an illustrative view of quantization and run lengthencoding;

[0053]FIG. 25 is a block diagram showing the constitution of aconventional image decoding apparatus based on an MPEG1 image decodingsystem; and

[0054]FIGS. 26A and 26B are illustrative views of OBMC (overlappedmotion compensation).

BEST MODES FOR CARRYING OUT THE INVENTION

[0055] Embodiments of the present invention will be described in detailhereinbelow by referring to the drawings.

[0056] (First Embodiment)

[0057]FIG. 1 is a block diagram showing the constitution of the imagecoding apparatus according to the first embodiment of the presentinvention, and FIG. 2 is a block diagram showing the constitution of theimage decoding apparatus. The image coding apparatus shown in FIG. 1performs efficient image coding on account of being constituted toreduce redundancy in a temporal direction by means of MC, quantifyspatial discontinuity between individual MC blocks with respect to theprediction image obtained by means of the MC, and adaptively performsmoothing filter processing in accordance with the conditions.

[0058] The MC procedure of this image coding apparatus is substantiallythe same as the method described in the conventional example. An outlineof this procedure is provided in FIG. 21 and an outline of the blockmatching processing employed in the motion vector detection is as shownin FIGS. 22A to 22D. However, the MC blocks can be defined by uniformlydivided units into which macroblocks are divided into a variety ofrectangular regions as shown by MC modes 1 to 7 in FIGS. 3A to 3G, andidentification data indicating which MC block shape is used istransmitted as coding mode data.

[0059] For example, in MC mode 1 shown in FIG. 3A, in order to make amacroblock into an MC block, one motion vector is established for themacroblock. On the other hand, in MC mode 2 shown in FIG. 3B, regionsproduced by dividing the macroblock into lateral halves are MC blocks,meaning that two motion vectors per macroblock are established.Similarly, in MC mode 7 shown in FIG. 3G, 16 motion vectors permacroblock are established.

[0060] A procedure that involves performing an orthogonal transform withrespect to a residual signal obtained from the difference between aninput image and a prediction image which has undergone smoothingprocessing, and then quantizing and entropy coding the correspondingcoefficients, is also as described with reference to FIG. 24 in theconventional example.

[0061] The operation of the image coding apparatus and image decodingapparatus shown in FIGS. 1 and 2 respectively will be described belowwith reference to the drawings and will be centered on the smoothingfilter processing which is a particular feature of the presentembodiment.

[0062] The operation of the image coding apparatus will be describedfirst. The input image signal 101 is a temporal array of frame imagesand will subsequently embody the signal of a frame image unit. A frameimage that is to be encoded is the current frame 601 shown in FIG. 21.

[0063] The current frame is encoded by means of the following procedure.The input image signal 101 is inputted to a motion detection section 102one macroblock at a time, and motion vectors 105 are detected in themotion detection section 102. Of the macroblock forms shown in FIGS. 3Ato 3G, the form which affords the best coding efficiency is selected forthe MC block which is the motion-vector assigned unit. The motioncompensation section 107 uses the motion vectors 105 to reference areference image 104 (encoded and locally decoded frame image) which isstored in the frame memory 103 and then extracts a prediction image 106a for the macroblocks.

[0064] Although the motion detection section 102 and the motioncompensation section 107 perform processing for every one of themacroblocks, the signal for the difference with respect to the inputimage signal 101 (the prediction residual signal 108) is obtained withthe frame as the unit. That is, the motion vectors 105 of individualmacroblocks are maintained over the entire frame, whereby the predictionimage 106 a is constituted as a frame-unit image.

[0065] Next, smoothing processing between MC blocks of the predictionimage 106 a is performed in the smoothing filter section 124. Thedetails of this processing will be described in detail subsequently. Thesmoothed prediction image 106 b is subtracted from the input imagesignal 101 by the subtraction section 131, and, as a result, theprediction residual signal 108 is obtained. The prediction residualsignal 108 is converted into orthogonal transform coefficient data 110by the orthogonal transform section 109. A DCT is used for example inthe orthogonal transform. The orthogonal transform coefficient data 110passes through a quantization section 111, and is scanned and run-lengthencoded by the variable length coding section 113, before beingmultiplexed and transmitted in a compressed stream 114 by same.

[0066] Thereupon, coding mode data 123 that indicates whether intraframecoding or interframe coding has been performed, which is determined onemacroblock at a time, is also multiplexed. In an inter mode case, motionvectors 105 are multiplexed and transmitted in a compressed stream 114one macroblock at a time. Further, quantized coefficients 112 arelocally decoded via a reverse quantization section 115 and a reverseorthogonal transform section 116, and the decoded result is added to theprediction image 106 b by an addition section 132 to thereby generate adecoding image 117 which is the same as that on the image decodingapparatus side. The decoding image 117 is stored in the frame memory 103to be used as a reference image 104 for the prediction of the nextframe.

[0067] Next, the operation of the image decoding apparatus will bedescribed with reference to FIG. 2. After the compressed stream 114 hasbeen received by the image decoding apparatus, the variable lengthdecoding section 118 detects a sync word that represents the start ofeach frame, whereupon coding mode data 123, motion vectors 105 andquantized orthogonal transform coefficients 112 are decoded inmacroblock units. The motion vectors 105 are outputted to the motioncompensation section 107, and the motion compensation section 107extracts, as the prediction image 106 a, the image parts which havemoved to an extent equivalent to the motion vectors 105 from a framememory 122 (used in the same way as the frame memory 103), thisoperation being similar to the operation of the image coding apparatus.

[0068] The prediction image 106 a passes through the smoothing filtersection 124 and is then outputted as the smoothed prediction image 106b. The quantized orthogonal transform coefficients 112 are decoded via areverse quantization section 120 and a reverse orthogonal transformsection 121, and then added by an addition section 133 to the predictionimage 106 b to form the final decoded image 117. The decoded image 117is stored in the frame memory 122 and outputted to a display device (notshown) with predetermined display timing, whereby the image is playedback.

[0069] Next, the operation of the smoothing filter section 124 will bedescribed. First, the grounds for the need for the smoothing filter willbe described with reference to FIG. 4. FIG. 4 shows an input image 141,a prediction image 142 in which no discontinuity between MC blocks isgenerated as well as a prediction residual image 143 which uses the,prediction image 142, and a prediction image 144 in which discontinuitybetween MC blocks is generated as well as a prediction residual image145 which uses the prediction image 144. Motion vectors are detectedwith MC blocks as units by means of a motion detection algorithm of ageneral image coding system such a block matching algorithm or the like.

[0070] That is, all the pixels contained the MC blocks possess the samemovement amount. Generally, in block unit MC, motion vectors whichafford the greatest reduction in the prediction residual for the MCblocks are detected, meaning that no consideration is given to spatialcontinuity with adjoining MC blocks. For this reason, as shown in FIG.4, a discontinuous waveform 146 is sometimes produced between MC blocks.This discontinuous waveform 146 remains in the residual signal and is asignal subjected to encoding. Here, in order to exercise caution, theorthogonal transform itself is performed only within the above MC blockand therefore has no influence on the coding of the frame.

[0071] However, in cases where, in the prediction residual coding ofthis frame, such a specific waveform cannot be adequately encoded, thewaveform component remains in the local decoding image and appearswithin the MC blocks in the prediction image of subsequent frames. Thissometimes influences the coding efficiency of the prediction residualsignal. In a natural image, the boundaries of the original MC blocksshould be smoothly linked, and, based on this assumption, the processingperformed by the smoothing filter section 124 has as an object to obtaina prediction image which is close to being a natural image by smoothingany discontinuous waveforms present between MC blocks.

[0072] The constitution of the smoothing filter section 124 is shown inFIG. 5 and will now be described. First, a block activity levelcalculation section 125 determines a block activity level S(X) in unitsof fixed blocks X in the prediction image 106 a. The flow of theprocessing by the block activity level calculation section 125 is shownin FIG. 6. The determination of S(X) is based on the relationship withthe neighboring blocks as shown in FIG. 7. Here, the blocks A to E areunits for determining the block activity level and are not necessarilythe same as the MC blocks. For example, the fact that block A is onepart of a larger MC block 147 is shown. In other words, the blockactivity level is directed toward blocks of fixed size that bear norelation to the size of the MC blocks shown in FIGS. 3A to 3G. First,S(X) is set to a predetermined initial value S0 (zero, for example) overthe whole area of the frame. Then, when the coding mode data 123 ofmacroblocks including block C. indicate the intra mode, the blockactivity level S(X) is determined in accordance with the following Rule1 (step ST1).

[0073] (Rule 1)

[0074] (1) current S(A) is updated to max {S(A), SO+1}

[0075] (2) current S(B) is updated to max {S(B), S0+1}

[0076] (3) current S(C) is updated to max {S(C), S0+2}

[0077] (4) current S(D) is updated to max {S(D), S0+1}

[0078] (5) current S(E) is updated to max {S(E), SO+1}

[0079] Therefore, the block activity level in the vicinity of the blockto be intra coded is set high. The resolution of the prediction image inintra coding is generally lower than that of a prediction image producedby inter coding and hence the boundaries of blocks among macroblocks ofintra mode stand out easily. The provision of step ST1 is equivalent toraising the priority of smoothing processing in such regions.

[0080] A description will be provided next for the rule for setting S(X)when the coding mode data 123 of macroblocks including block C indicatesinter coding. First, it is judged whether or not the current predictionimage was generated using bidirectional prediction (the B frameprediction mentioned in the conventional example) (step ST2).

[0081] In cases where bidirectional prediction can be used, theprediction direction can be changed for each of the macroblocks. Whenthe prediction direction differs between blocks, spatial continuity atthe boundaries of both blocks cannot be assumed. That is, a judgment ismade of whether or not the prediction direction of blocks A, B, D, andE, which adjoin block C, is the same, and processing is then switched(step ST3).

[0082] When only unidirectional prediction is used or when the framepermits bidirectional prediction and the prediction direction for blockC is the same, the block activity level is updated in accordance withRule 2 below (step ST4).

[0083] (Rule 2)

[0084] (1) If macroblocks including block A are of an inter mode,current S(A) is updated to max {S(A), K} and

[0085] current S(C) is updated to max {S(C), K}.

Here, K=2 (when mvd(A,C)≧3)

K=1 (when 0<mvd(A,C)<3)

K=0 (when mvd(A,C)=0)

[0086] (2) If macroblocks including block B are of an inter mode,current S(B) is updated to max {S(B), K} and

[0087] current S(C) is updated to max {S(C), K}.

Here, K=2 (when mvd(B,C)≧3)

K=1 (when 0<mvd(B,C)<3)

K=0 (when mvd(B,C)=0)

[0088] (3) If macroblocks including block D are of an inter mode,current S(D) is updated to max {S(D), K} and

[0089] current S(C) is updated to max {S(C), K}.

Here, K=2 (when mvd(D,C)≧3)

K=1 (when 0<mvd(D,C)<3)

K=0 (when mvd(D,C)=0)

[0090] (4) If macroblocks including block E are of an inter mode,current S(E) is updated to max {S(E), K} and

[0091] current S(C) is updated to max {S(C), K}.

Here, K=2 (when mvd(E,C)≧3)

K=1 (when 0<mvd(E,C)<3)

K=0 (when mvd(E,C)=0)

[0092] (5) If blocks A, B, D, and E are intra coded, the block activitylevel thereof is not changed.

[0093] In the above rule, mvd (X,Y) indicates the large value of thedifferential values for the components of motion vectors of adjoiningblocks X and Y. Further, max (a,b) indicates the larger value of thevalues a and b. By updating the above block activity level, a high blockactivity level can be provided between blocks exhibiting a marked motionvector difference.

[0094] When mvd (X, Y)=0 (when there is no motion vector differencebetween blocks X and Y), this represents a case where the block boundaryretains complete spatial continuity and there is thus no need forsmoothing here. The block activity level is therefore set to a minimumvalue.

[0095] On the other hand, in a frame permitting the use of bidirectionalprediction and when the direction of the prediction for the blocks A, B,D, E is different in relation to block C, or in a mode that combinesprediction images by adding and averaging the prediction values inforward and backward directions, the spatial continuity of theprediction image is broken irrespective of the motion vector difference,and hence the current S(X) is updated (step ST5) to max {S(X),1} (Xrepresents blocks A to E). The above processing is performed untilcompleted for all the fixed blocks X in the frame (step ST6), and thesetting of the block activity level S(X) 126 is thus completed.

[0096] By using the block activity level S (X) 126 set by the blockactivity level calculation section 125, smoothing processing between MCblocks is performed for the prediction image 106 a in the filterprocessing section 127. In the filter processing process, futile filterprocessing is corrected in the post-processing section 129 so that theresults 128 of performing smoothing filtering once do not produceexcessive smoothing. The process for this processing is shown in theflowchart of FIG. 8, and the state of pixels to be processed betweenlaterally adjoining blocks is shown in FIG. 9 and will now be described.

[0097] That is, in FIG. 9, the pixels r₁ to r₃ are contained in block nand the pixels l₁ to l₃ are contained in block n−1 which adjoins block non the left. In the description that follows, the assumption is madethat block n and block n−1 possess mutually different vectors MV (n) andMV (n−1) respectively and that an MC block boundary BD lies between r₁and l₁. The definition is also the same for pixels to be processedbetween vertically adjoining blocks.

[0098] First, it is judged whether or not the magnitude of the blockboundary difference value d=|r₁−l₁| (here, r₁, l₁ represent the pixelvalues of the respective pixels r₁, l₁) exceeds a threshold value α (S)established in accordance with the block activity level S (step ST7). Inthe filter processing below, processing is performed for two blockboundaries and therefore, the larger value of the values of S (X) forthe two blocks processed is used as the block activity level S. Forexample, when filtering is performed on the boundaries of block B andblock C in FIG. 7, if S(B)>S(C), the value of S(B) is used as the blockactivity level S. When the differential value d is equal to or less thanthe threshold value α (S), filter processing is not performed in thepixel regions of r₁ to r₃, and l₁ to l₃. On the other hand, when thedifferential value d exceeds the threshold value α (S), filterprocessing is performed by switching the pixel region to be filtered, inaccordance with S (step ST8).

[0099] As a result, if S=0, there is no discontinuity at the blockboundaries and so the filter processing is skipped. If S=1, filterprocessing is performed for the two pixels which are pixels r₁ and l₁(step ST9). As shown in FIG. 9, for example, there are methods includingone where the filter processing employs a low pass filter F constitutedto use three points, which are the pixels r₁, l₁, and r₂, for pixel r₁,and to use three points, which are the pixels l₁, l₂, and r₁, for pixell₁, but any given filter can be used. For example, a 5-tap filter, whichhas the pixels r₁ or l₁ at the center and which uses two pixels to theleft and right (or above and below) thereof respectively, may beemployed. As a further example of a filter, constitutions may also beconsidered such as a constitution in which, as shown in FIG. 10A, whenpixel l₁ is filtered, pixel r₁ and prediction pixel value lr₁ of thepositions of pixel l₁ which is extracted by means of the vector MV (n)of pixel r₁, is employed, and, as shown in FIG. 10B, when pixel r₁ isfiltered, pixel l₁ and prediction pixel value rl₁ of the positions of r₁which is extracted by means of the vector MV (n−1) of pixel l₁, isemployed. The prediction pixel value lr₁ and rl₁ are pixel values whichare spatially linked from the start to the pixels r₁ and l₁ in areference image region in the frame memory, which makes more naturalsmoothing of block boundaries possible.

[0100] When S=2, in addition to the pixels r₁ and l₁ pixels r₂ and l₂are pixels that are targeted for smoothing (steps ST10 and ST11). Incases where S=2, there are often steep and discontinuous boundaries dueto the high block activity level, and hence the object is to increasethe continuity of the signal by increasing the extent of the smoothing.

[0101] The above processing is carried out in a filter processingsection 127. The prediction pixel value 128 produced by filterprocessing is corrected so as to be effective in the coding efficiencyof a post-processing section 129. The processing of the post-processingsection 129 is equivalent to steps ST12 and ST13 in FIG. 8. Thepost-processing section 129 controls the differential value Δ betweenthe pixel value before filtering and the pixel value after filtering bymeans of a threshold value Th(S) established in accordance with theblock activity level S.

[0102] Specifically, the functions shown in FIG. 11 (horizontal axis: Δ,vertical axis: Δ correction value) are established and the correctionvalue is determined. Here, the threshold value Th(S) is the maximumpermissible differential value, and in cases where a Δ that is equal toor more than this value is produced, correction is applied in thedifference-diminishing direction in accordance with the size of thisvalue. In cases where Δ is equal to or more than the threshold valueTh(S), the assumption is made that the difference obtained by filteringis not attributable to MC block discontinuity but instead is the resultof filtering with respect to an edge component that was originallypresent in the image.

[0103] Therefore, according to the image coding apparatus and imagedecoding apparatus according to the first embodiment, it is possible toperform correction in the direction in which only filtering to correctMC block discontinuity is allowed, by means of the above correctivemeasures performed by the smoothing filter section 124. The predictionimage 106 b is outputted via the above processing and it is thereforepossible to improve the coding efficiency by suppressing discontinuouswaveforms generated in the prediction residual.

[0104] When setting the block activity level in FIG. 7, in inter mode,the block activity level is determined up to correspond to the range ofvalues of mvd (X, Y), but the method used to determine this range isoptional. More particularly, the block activity level in an inter modecase may be determined using only the criterion that mvd (X, Y) shouldbe zero or not zero. Further, since it can be said that the smaller themotion vector-assigned units in the variety of MC block forms shown inFIGS. 3A to 3G become and the larger the number of motion vectors permacroblock becomes, the more intense the motion in the macroblock and inthe vicinity thereof becomes, the block activity level may be set basedon the criterion that any of the MC modes 1 to 7 shown in FIGS. 3A to 3Gshould be selected.

[0105] Furthermore, this smoothing filter processing can also beconstituted so that same can be turned ON/OFF in frame units. Theprocessing itself of the smoothing filter section 124 is processing tochange prediction image data selected optimally in MC block units, andtherefore this processing can also have an adverse effect as well as agood effect on the coding efficiency. Thus, image analysis in frameunits is performed by the image coding apparatus. It is judgedbeforehand whether or not motion causing discontinuity between MC blocksis present, and the smoothing filter section 124 is turned ON whendiscontinuity is generated and turned OFF in the absence ofdiscontinuity.

[0106] Examples of image analysis include the evaluation of aprovisional residual between the input image signal 101 and theprediction image 106 a. The signal distribution of the residual isviewed and, because residual coding processing does not requiresmoothing filter processing in frames that are not particularlydisadvantageous, the filter is turned OFF, but the filter is turned ONfor frames that are significantly disadvantageous. For example,consideration may be given to operation such that in cases where theproportion of the residual signal amount at the MC boundaries inrelation to the overall residual signal amount is equal to or more thana certain fixed threshold value, the filter is turned ON, and when thisproportion is equal to or less than a threshold value, the filter isturned OFF. Alternatively, there are also methods in which adetermination of whether to turn the filter ON or OFF is made after theframe unit coding efficiency has been compared in cases where smoothingprocessing is and is not performed. The result of the ON/OFFdetermination is transmitted as a portion (bit data that indicates thepresence or absence of smoothing) of the header information of the startof a frame in the compressed stream 114. By means of such aconstitution, smoothing processing can be applied more adaptively for anirregular image signal.

[0107] (Second Embodiment)

[0108]FIG. 12 is a block diagram showing the constitution of the imagecoding apparatus according to the second embodiment of the presentinvention, and FIG. 13 is a block diagram showing the constitution ofthe image decoding apparatus thereof. The second embodiment relates toan apparatus constituted by introducing the smoothing filter of thepresent invention described above to an image coding and decodingapparatus according to a compression coding system that applies thetechnique known as Matching Pursuits. Image coding systems that useMatching Pursuits include that disclosed by R. Neff et al, “Very LowBit-rate Video Coding Based on Matching Pursuits”, IEEE Trans. on CSVT,vol. 7, pp. 158-171, February 1997. With Matching Pursuits, a predictionresidual image signal f to be encoded can be rendered as per thefollowing formula by using an over-complete basis set G prepared inadvance that comprises n types of basis g_(k)ε G (1≦k≦n) $\begin{matrix}{f = {\left( {\sum\limits_{i = 0}^{m - 1}{{\langle{s_{i\quad},g_{ki}}\rangle}g_{ki}}} \right) + r_{m}}} & (1)\end{matrix}$

[0109] Here, m is the total number of basis search steps, i is the basissearch step number, and r_(i) is the prediction residual image signalfollowing completion of the basis search of the (i−1)th step, thissignal being without further processing the prediction residual imagesignal for the basis search of the ith step, where r₀=f. Further, s_(i)and g_(ki) are the partial region and basis respectively, these beingobtained by selecting, in the basis search of the ith step, acombination of s and g_(k) such that the inner product value thereof ismaximized, from optional partial regions s (partial regions in a frame)of r_(i), as well as optional bases g_(k) contained in the basis set G.If the basis search is performed thus, the larger the number m of basissearch steps, the less energy r_(m) diminishes. This means that thegreater the number of bases used in the rendition of the predictionresidual image signal f, the better the signal can be rendered.

[0110] In each of the basis search steps, the data that is encoded is:

[0111] 1) The index expressing g_(ki) (g_(k) is shared and maintained onthe encoding side and the decoding side, which makes it possible tospecify a basis by converting only the index data).

[0112] 2) The inner product values <s_(i), g_(ki)> (correspond to thebasis coefficients), and

[0113] 3) s_(i) on-screen center position data p_(i)=(x_(i), y_(i))

[0114] A set of these parameters is collectively known as an atom. Bymeans of this image signal rendition and encoding method, the number ofencoded atoms is increased, that is, as the total number m of basissearch steps increases, so too does the encoded volume, wherebydistortion is reduced.

[0115] On the other hand, according to the image coding performed byMatching Pursuits in the above paper, MC is carried out independentlyfrom Matching Pursuits, and atom extraction is performed with respect tothe prediction residual signal. In this case, there is the possibilitythat atoms will be extracted in positions extending over the MC block.So long as a system is adopted in which MC is dependent on the blockstructure, there is the disadvantage that a discontinuous waveformbetween MC blocks as described in the first embodiment above remains inthe residual signal and thus a waveform which should not be encoded willbe encoded.

[0116] Conventionally, overlapped MC that considers the motion vectorsof neighboring MC blocks has been utilized as a measure to resolve theforegoing problem. However, overlapped MC references more numerousprediction values and performs calculations for the final predictionvalue by means of a weight sum and there is therefore the problem thatthe computational cost is high and it is not possible to performadaptive smoothing with respect to the pixel values in the MC blocks,which obscures the prediction image excessively. By performing adaptivesmoothing filter processing at the MC block boundaries as described inthe first embodiment, smoothing of the residual signal can be performedwithout obscuring the prediction image excessively.

[0117] In the image coding apparatus shown in FIG. 12, the input imagesignal 201 is a temporal array of frame images which will subsequentlyembody the frame image unit signal. However, the frame image to beencoded corresponds to the current frame 601 shown in FIG. 21. Thecurrent frame is encoded by means of the following procedure.

[0118] First of all, the current frame is outputted to a motiondetection section 202, and detection of the motion vectors 205 isperformed by means of a procedure that is exactly the same as that ofthe motion detection section 102 of the first embodiment above. However,the motion detection section 202 divides the intra coding into that forthe DC component and that for the AC component. The result of encodingthe DC component is used as part of the prediction image and the ACcomponent is encoded as part of the prediction residual. Thisconstitutes processing to obtain the prediction image batchwise in frameunits in order to use the Matching Pursuits.

[0119] Accordingly, when intra mode is selected in the motion detectionsection 202, the corresponding macroblock prediction image is filled byan intra DC component which is encoded and locally decoded. The intra DCcomponent undergoes prediction from neighboring image data as well asquantization in a DC coding section 225, and is outputted to a variablelength decoding section 213 as encoded data 226 and multiplexed in acompressed stream 214.

[0120] A motion compensation section 207 uses the DC component as abovefor intra mode macroblocks, and, for inter mode macroblocks, uses motionvectors 205 to reference a local decoding image 204 in the frame memory203, whereby a prediction image 206 a for the current frame is obtained.Although the motion detection section 202 and the motion compensationsection 207 perform processing for each of the macroblocks, thedifferential signal with respect to the input image signal 201 (theprediction residual signal 208) is obtained by taking the frame as theunit. That is, the motion vectors 205 of individual macroblocks aremaintained over the entire frame, whereby the prediction image 206 a isconstituted as a frame-unit image.

[0121] Next, the smoothing filter section 224 performs smoothingprocessing between the MC blocks of the prediction image 206 a. Theoperation of a smoothing filter section 224 uses coding mode data 223and motion vectors 205 and is implemented by means of processing likethat in the first embodiment. The smoothed prediction image 206 b issubtracted from the input image signal 201 by a subtraction section 241to obtain a prediction residual signal 208.

[0122] Next, the atom extraction section 209 generates atom parameters210 on the basis of the above-described Matching Pursuits algorithm,with respect to the prediction residual signal 208. A basis set g_(k)211 is stored in a basis codebook 210. If, based on the properties ofthe Matching Pursuits algorithm, a basis which can render the partialsignal waveform as accurately as possible can be found in an initialsearch step, the partial signal waveform can be rendered by fewer atoms,that is, with a small encoded volume. Atoms are extracted over the wholearea of the frame. For the coding of the position data in the atomparameters, making use of the fact that the atom coding order does notinfluence the decoding image, sorting is performed such that the atomsare aligned in order using two-dimensional co-ordinates with the topleft-hand corner of the frame as the starting point, and the codingorder is constructed so that the atoms are counted in macroblock units.The macroblock units are therefore constituted such that atom parameters212 (the respective basis index, position data, and basis coefficient)are coded in proportion to the number of atoms contained in themacroblock units.

[0123] An atom decoding section 215 decodes a local decoding residualsignal 216 from the atom parameters 212 and then obtains a localdecoding image 217 by adding the local decoding residual signal 216 tothe smoothed prediction image 206 b by means of an addition section 242.The local decoding image 217 is stored in the frame memory 203 in orderto be used in the MC for the next frame.

[0124] Next, the image decoding apparatus will be described by referringto FIG. 13. After the compressed stream 214 has been received by theimage decoding apparatus, a variable length decoding section 229 detectsa sync word that indicates the start of each frame, whereupon codingmode data 223, motion vectors 205 and atom parameters 212 are decoded inmacroblock units. The motion vectors 205 are outputted to the motioncompensation section 207 and the output 206 a is inputted to thesmoothing filter section 224, whereby the prediction image 206 b isobtained. The atom parameters 212 are decoded by the atom decodingsection 215. A basis is extracted by supplying a basis index to thebasis codebook 210. The output 216 of the atom decoding section 215 isadded to the prediction image 206 b by means of an addition section 243to produce the decoding image 217. The decoding image 217 is used in theMC for subsequent frames, and is therefore stored in a frame memory 230.The decoding image 217 is outputted to a display device (not shown) withpredetermined display timing, whereby the image is played back.

[0125] Therefore, according to the image coding apparatus and the imagedecoding apparatus of the second embodiment, results similar to thosefor the first embodiment above can be obtained also for an image codingand decoding apparatus according to a compression coding system thatapplies the technique known as Matching Pursuits.

[0126] (Third Embodiment)

[0127] A third embodiment of the present invention will now bedescribed. The third embodiment describes another smoothing filtersection. This smoothing filter section is a modification of thesmoothing filter sections 124 and 224 described in the above first andsecond embodiments respectively, and because this filter simplysubstitutes for the smoothing filter sections 124 and 224, this filtercan be applied to the image coding apparatus and image decodingapparatus shown in FIGS. 1 and 2 or FIGS. 12 and 13 respectively. Theinternal constitution is also the same as that in FIG. 5.

[0128] With the smoothing filter section according to the thirdembodiment, the block activity level calculation section 125 does notdefine the block activity level information with respect to the blocksbut instead defines this information with respect to the blockboundaries. Consequently, the filter can be controlled by uniquelyallocating an activity level without the selection of an activity levelused in circumstances where the activity level differs between blocks aswas described in the first and second embodiments.

[0129] The activity level is defined with respect to block boundariesand therefore, as shown in FIG. 14, activity levels S_(L)(C) andS_(U)(C) are defined for two boundaries to the left and above one block(‘C’ here) respectively. In the determination of S_(L)(C), the activitylevel is found based on the relationship with block B to the left, andwhen S_(U)(C) is determined, the activity level is found based on therelationship with block A above.

[0130] The boundaries between blocks D and E are determined as S_(L)(D)and S_(U)(E) respectively. As indicated in the first embodiment, themethod of determining the activity level is determined by the motionvector difference between two blocks and by a difference in coding modetherebetween, and can therefore be determined using setting rules likethose for the first embodiment.

[0131] Therefore, according to the smoothing filter section of the thirdembodiment, the filter can be controlled by uniquely allocating anactivity level without the selection of an activity level used incircumstances where the activity level differs between blocks as wasdescribed in the first and second embodiments.

[0132] Furthermore, in the third embodiment, because the determinationof the activity level is dependent on the blocks to the left and abovealone, the apparatus, which generates the prediction image in macroblockunits and encodes and decodes this image, is also able to carry outencoding and decoding processing while performing smoothing of the MCblocks. Further, by introducing pipeline processing in macroblock units,implementation that enables rapid and efficient processing is possiblefor the image coding apparatus and image decoding apparatus.

[0133] (Fourth Embodiment)

[0134] A fourth embodiment of the present invention will now bedescribed. The fourth embodiment describes another smoothing filtersection. This smoothing filter section is a modification of thesmoothing filter sections 124 and 224 described in the above first andsecond embodiments respectively, and because this filter simplysubstitutes for the smoothing filter sections 124 and 224, this filtercan be applied to the image coding apparatus and image decodingapparatus shown in FIGS. 1 and 2 or FIGS. 12 and 13 respectively. Theinternal constitution is also the same as that in FIG. 5.

[0135] The smoothing filter section of the fourth embodiment switchesthe filter characteristics in accordance with the activity level. FIGS.15A and 15B illustrate a case where a 5-tape filter which has the pixelr₁ to be filtered at the center and which uses two pixels to the leftand right thereof respectively, is applied. As shown in FIG. 15B, in acase where the activity level is high (S=2) and there is a desire toraise the extent of the smoothing still further, a filter that increasesthe influence of the nearby pixels in the filter window is applied, andconversely, in a case where the activity level is low (S=1) and there isa desire to suppress the excessive loss of detail caused by smoothing, afilter with which the pixel itself has considerable influence isapplied, as shown in FIG. 15A.

[0136] Therefore, according to the smoothing filter section of thefourth embodiment, the extent of the smoothing according to the activitylevel can be controlled.

[0137] Further, the constitution may be such that, in the switching ofthe filter characteristics, a plurality of characteristics can beselected in accordance with the activity level, and such thatinformation identifying the characteristics is multiplexed in acompressed stream 114 and transmitted to the image decoding apparatus.By mean of such a constitution, a more detailed adaptive judgment basedon image analysis on the image coding apparatus side can be reflected inthe filter characteristics, and the image decoding apparatus can thusimplement adaptive smoothing filter processing without performingspecial image analysis processing as implemented by the image encodingapparatus. The fourth embodiment is equally applicable in cases of usingan activity level which is defined for block boundaries as described inthe third embodiment.

[0138] When the filter characteristics are switched, the type of filtercharacteristics used is transmitted as part of the header information atthe start of the frame in the compressed stream, for example.

[0139] (Fifth Embodiment)

[0140]FIG. 16 is a block diagram showing the constitution of the imagecoding apparatus according to the fifth embodiment of the presentinvention, and FIG. 17 is a block diagram showing the constitution ofthe image decoding apparatus thereof. However, those parts of the imagecoding apparatus of the fifth embodiment shown in FIG. 16 whichcorrespond to parts of the first embodiment in FIG. 1 have been assignedthe same reference symbols, and those parts of the image decodingapparatus of the fifth embodiment shown in FIG. 17 which correspond toparts of the first embodiment in FIG. 2 have been assigned the samereference symbols, and hence a description of all these parts is omittedhere.

[0141] The image coding apparatus shown in FIG. 16 and the imagedecoding apparatus shown in FIG. 17 differ from those of the firstembodiment insofar as another smoothing filter section 524 is employedin place of the smoothing filter section 124. The internal constitutionof the smoothing filter section 524 is shown in FIG. 18 and a flowchartillustrating the operation of a block activity level calculation section525 in the smoothing filter section 524 is shown in FIG. 19.

[0142] That is, this smoothing filter section 524 is a modification ofthe smoothing filter sections 124 and 224 described in the above firstand second embodiments respectively, and, with the exception of theinputting of the reference image 104, can simply substitute for thesefilters. In the fifth embodiment, the difference is obtained between theprediction image 106 a prior to smoothing filter processing, and thereference image 104 which is in the frame memory 103 and from which theprediction image 106 a originated, and filter control is performed onthe basis of the corresponding error margin electric power.

[0143] The prediction image 106 a is image data extracted from thereference image 104 using the motion vectors 105, and is image data thatapproximates the input image signal 101 inputted to the image codingapparatus. In other words, when points that are spatially the same inthe reference image 104 and the prediction image 106 a are compared, theerror margin electric power is large in parts with movement, and inparts with very little movement, the error margin electric power issmall. The magnitude of the motion vectors 105 does to some extentexpress the movement amount, but primary factors that are not dependenton a change to the image, such as noise, also influence detection, andtherefore the extent and intensity of the movement cannot be adequatelyexpressed by this magnitude alone. However, the above error marginelectric power can be used as an indicator for the intensity of themovement, whereby the adaptability of the filter control can beimproved. Further, the reference image 104 can use exactly the same dataon the encoding and decoding sides and therefore, when introducing thiscontrol, implementation is possible without transmitting specialidentification information to the decoding apparatus.

[0144] Specifically, as shown in FIG. 18, a reference image 104 and aprediction image 106 a are inputted to the block activity levelcalculation section 525, and the error margin electric power between thereference image 104 and the prediction image 106 a is found for eachblock. As shown in step ST14 in FIG. 19, in order to reduce the amountof extra computation here, the evaluation based on the error marginelectric power is skipped at points where it is judged that the activitylevel is zero and there is no movement. This is because in cases wherethe motion vector difference mvd (X, Y) is zero, spatial continuity ismaintained even though this may be a part with intense movement, meaningthat it is not necessary to perform smoothing.

[0145] In cases where the activity level at least is more than zero, theactivity level is not evaluated by means of motion vectors alone.Instead, the error margin electric power thus found is used, and whensame is greater than a predetermined threshold value, the activity levelis changed toward a larger value, and when the error margin electricpower is smaller than a predetermined threshold value, the activitylevel is set to zero and smoothing is not performed (step ST15). At suchtime, the threshold value in the direction of raising the activity leveland the threshold value in the direction of lowering the activity levelneed not necessarily be the same.

[0146] Further, in the fifth embodiment, as far as the reference image104 is concerned, the constitution may be such that average values inblocks are precalculated and buffered in evaluation block units beforethis image is stored in the frame memory, and average values aresimilarly found for the prediction image 106 a, whereby the error marginelectric power evaluation is performed using only average values.

[0147] Because the average values of the error margin amounts betweenthe reference image 104 and the prediction image 106 a are controllingcomponents and the average values alone can be stored in a small buffer,the frequency of access to the frame memory during an activity levelcalculation can be reduced without affecting the judgment of theactivity level.

[0148] Furthermore, when the activity level is allocated at the blockboundaries as is the case in the third embodiment above, theconstitution can also be such that partial regions that extend acrossthe block boundaries are defined and the error margin amount between thereference image 104 and the prediction image 106 a is evaluated in theseunits.

[0149] In addition, in the fifth embodiment, an error margin amountbetween the reference image 104 and the prediction image 106 a is usedto update the activity level but may also be used to change the filtercharacteristics applied for points possessing a certain predeterminedactivity level value. For example, in cases where the activity level ofa certain block or block boundary is an intermediate value in a definedactivity level range, as the filter characteristics at this time arechanged in accordance with the conditions, the adaptability increasesstill further. In order to achieve this object, a constitution is alsopossible in which an evaluation is adopted in which the error marginamount between the reference image 104 and the prediction image 106 a isswitched.

[0150] Therefore, by means of the smoothing filter section of the fifthembodiment, the adaptability of the filter control can be improved asdescribed above, and the reference image 104 is able to use exactly thesame data on the encoding and decoding sides, and therefore, whenintroducing this control, implementation is possible withouttransmitting special identification information to the decodingapparatus. Further, the frequency of access to the frame memory duringan activity level calculation can be reduced without affecting thejudgment of the activity level.

[0151] Industrial Applicability

[0152] The present invention can be used as an image coding apparatusand an image decoding apparatus applied to a mobile image transmissionsystem, for example.

1. An image coding apparatus, characterized by comprising: motioncompensation predicting means for generating a motion-compensatedprediction image by detecting movement amounts in predetermined partialimage region units of an input image; smoothing means for performingsmoothing of pixels located at the boundaries of adjoining partial imageregions on the basis of predetermined evaluation criteria, with respectto the prediction image obtained by the motion compensation predictingmeans; and prediction residual coding means for coding the predictionresidual signal obtained from the difference between the input image andthe smoothed prediction image.
 2. The image coding apparatus as definedin claim 1, characterized in that the motion compensation predictingmeans detect the movement amount by using encoded local decoding imagedata as a reference image.
 3. The image coding method as defined inclaim 1, characterized in that the smoothing means perform the smoothingin accordance with a difference value for the movement amount betweenthe adjoining partial image regions.
 4. The image coding apparatus asdefined in claim 1, characterized in that the smoothing means performthe smoothing in accordance with the difference in coding method betweenthe adjoining partial image regions.
 5. The image coding apparatus asdefined in claim 1, characterized in that the smoothing means performthe smoothing in accordance with the difference in image predictionmethod between the adjoining partial image regions.
 6. The image codingapparatus as defined in claim 1, characterized in that the smoothingmeans perform the smoothing in accordance with the error margin amountbetween the prediction image prior to smoothing, and the reference imagefrom which the prediction image obtained by the motion compensationpredicting means is generated.
 7. The image coding apparatus as definedin claim 1, characterized in that the smoothing means comprise: activitylevel setting means for determining the activity level between partialimage regions which is determined on the basis of at least one of: adifference value for the movement amount between the adjoining partialimage regions, a difference in coding method between the adjoiningpartial image regions, a difference in image prediction method betweenthe adjoining partial image regions, and an error margin amount betweenthe prediction image prior to smoothing, and the reference image fromwhich the prediction image obtained by the motion compensationpredicting means is generated; adaptive smoothing means for establishingthe intensity of the smoothing on the basis of the activity level andthen performing smoothing of pixels located at the boundaries betweenthe partial image regions; and post-processing means for performingsuppression processing, of the results of smoothing, by means of athreshold value established in accordance with the activity level. 8.The image coding apparatus as defined in claim 7, characterized in thatthe activity level setting means set the activity level for individualpartial image regions and input to the adaptive smoothing means thelarger value of values for the activity level obtained by comparingpartial image regions which are subjected to smoothing.
 9. The imagecoding apparatus as defined in claim 7, characterized in that theactivity level setting means set the activity level for boundariesbetween the partial image regions and input the activity level to theadaptive smoothing means.
 10. The image coding apparatus as defined inclaim 7, characterized in that the adaptive smoothing means change thenumber of pixels subjected to smoothing in accordance with the activitylevel obtained by the activity level setting means.
 11. The image codingapparatus as defined in claim 7, characterized in that the adaptivesmoothing means switch filter characteristics for performing thesmoothing in accordance with the activity level obtained by the activitylevel setting means.
 12. The image coding apparatus as defined in claim11, characterized in that the adaptive smoothing means encode andtransmit a bit that indicates the type of filter characteristics forperforming the smoothing.
 13. The image coding apparatus as defined inclaim 1, characterized in that a bit that indicates the presence orabsence of smoothing by the smoothing means is encoded and transmitted.14. An image decoding apparatus, characterized by comprising: motioncompensation predicting means for generating a motion-compensatedprediction image by detecting movement amounts in predetermined partialimage region units; smoothing means for performing smoothing of pixelslocated at the boundaries of adjoining partial image regions on thebasis of predetermined evaluation criteria, with respect to theprediction image obtained by the motion compensation predicting means;prediction residual decoding means for decoding a prediction residualsignal from the encoding side; and adding means for obtaining a decodedimage by adding together a decoded prediction residual signal obtainedby the prediction residual decoding means, and the smoothed predictionimage.
 15. The image decoding apparatus as defined in claim 14,characterized in that the motion compensation predicting means obtainthe prediction image by using decoded local decoding image data as areference image.
 16. The image decoding apparatus as defined in claim14, characterized in that the smoothing means perform the smoothing inaccordance with a difference value for the movement amount between theadjoining partial image regions.
 17. The image decoding apparatus asdefined in claim 14, characterized in that the smoothing means performthe smoothing in accordance with the difference in decoding methodbetween the adjoining partial image regions.
 18. The image decodingapparatus as defined in claim 14, characterized in that the smoothingmeans perform the smoothing in accordance with the difference in imageprediction method between the adjoining partial image regions.
 19. Theimage decoding apparatus as defined in claim 14, characterized in thatthe smoothing means perform the smoothing in accordance with the errormargin amount between the prediction image prior to smoothing, and thereference image from which the prediction image obtained by the motioncompensation predicting means is generated.
 20. The image decodingapparatus as defined in claim 14, characterized in that the smoothingmeans comprise: activity level setting means for determining theactivity level between partial image regions which is determined on thebasis of at least one of: a difference value for the movement amountbetween the adjoining partial image regions, a difference in codingmethod between the adjoining partial image regions, a difference inimage prediction method between the adjoining partial image regions, andan error margin amount between the prediction image prior to smoothing,and the reference image from which the prediction image obtained by themotion compensation predicting means is generated; adaptive smoothingmeans for establishing the intensity of the smoothing on the basis ofthe activity level and then performing smoothing of pixels located atthe boundaries between the partial image regions; and post-processingmeans for performing suppression processing, of the results ofsmoothing, by means of a threshold value established in accordance withthe activity level.
 21. The image decoding apparatus as defined in claim20, characterized in that the activity level setting means set theactivity level for individual partial image regions and input to theadaptive smoothing means the larger value of values for the activitylevel obtained by comparing partial image regions which are subjected tosmoothing.
 22. The image decoding apparatus as defined in claim 20,characterized in that the activity level setting means set the activitylevel for boundaries between the partial image regions and input theactivity level to the adaptive smoothing means.
 23. The image decodingapparatus as defined in claim 20, characterized in that the adaptivesmoothing means change the number of pixels subjected to smoothing inaccordance with the activity level obtained by the activity levelsetting means.
 24. The image decoding apparatus as defined in claim 20,characterized in that the adaptive smoothing means switch filtercharacteristics for performing the smoothing in accordance with theactivity level obtained by the activity level setting means.
 25. Theimage decoding apparatus as defined in claim 24, characterized in thatthe adaptive smoothing means switch the filter characteristics on thebasis of a bit that indicates the type of filter characteristics forperforming the smoothing, this bit being decoded from compressed inputdata from the encoding side.
 26. The image decoding apparatus as definedin claim 14, characterized in that the smoothing processing iscontrolled on the basis of a bit that indicates the presence or absenceof the smoothing, this bit being decoded from the compressed input data.27. An image coding method, comprising: a motion compensation predictingstep of generating a motion-compensated prediction image by detectingmovement amounts in predetermined partial image region units of an inputimage; a smoothing step of performing smoothing of pixels located at theboundaries of adjoining partial image regions on the basis ofpredetermined evaluation criteria, with respect to the prediction imageobtained by the motion compensation predicting step; and a predictionresidual coding step of coding the prediction residual signal obtainedfrom the difference between the input image and the smoothed predictionimage.
 28. The image coding method as defined in claim 27, characterizedin that the motion compensation predicting step detects the movementamount by using encoded local decoding image data as a reference image.29. The image coding method as defined in claim 27, characterized inthat the smoothing step performs the smoothing in accordance with adifference value for the movement amount between the adjoining partialimage regions.
 30. The image coding method as defined in claim 27,characterized in that the smoothing step performs the smoothing inaccordance with the difference in coding method between the adjoiningpartial image regions.
 31. The image coding method as defined in claim27, characterized in that the smoothing step performs the smoothing inaccordance with the difference in image prediction method between theadjoining partial image regions.
 32. The image coding method as definedin claim 27, characterized in that the smoothing step performs thesmoothing in accordance with the error margin amount between theprediction image prior to smoothing, and the reference image from whichthe prediction image obtained by the motion compensation predicting stepis generated.
 33. The image coding method as defined in claim 27,characterized in that the smoothing step comprises: an activity levelsetting step of determining the activity level between partial imageregions which is determined on the basis of at least one of: adifference value for the movement amount between the adjoining partialimage regions, a difference in coding method between the adjoiningpartial image regions, a difference in image prediction method betweenthe adjoining partial image regions, and an error margin amount betweenthe prediction image prior to smoothing, and the reference image fromwhich the prediction image obtained by the motion compensationpredicting means is generated; an adaptive smoothing step ofestablishing the intensity of the smoothing on the basis of the activitylevel and then performing smoothing of pixels located at the boundariesbetween the partial image regions; and a post-processing step ofperforming suppression processing, of the results of smoothing, by meansof a threshold value established in accordance with the activity level.34. The image coding method as defined in claim 33, characterized inthat the activity level setting step sets the activity level forindividual partial image regions and input to the adaptive smoothingstep the larger value of values for the activity level obtained bycomparing partial image regions which are subjected to smoothing. 35.The image coding method as defined in claim 33, characterized in thatthe activity level setting step sets the activity level for boundariesbetween the partial image regions and input the activity level to theadaptive smoothing step.
 36. The image coding method as defined in claim33, characterized in that the adaptive smoothing step changes the numberof pixels subjected to smoothing in accordance with the activity levelobtained by the activity level setting step.
 37. The image coding methodas defined in claim 33, characterized in that the adaptive smoothingstep switches filter characteristics for performing the smoothing inaccordance with the activity level obtained by the activity levelsetting step.
 38. The image coding method as defined in claim 37,characterized in that the adaptive smoothing step encodes and transmitsa bit that indicates the type of filter characteristics for performingthe smoothing.
 39. The image coding method as defined in claim 27,characterized in that a bit that indicates the presence or absence ofsmoothing by the smoothing step is encoded and transmitted.
 40. An imagedecoding method, characterized by comprising: a motion compensationpredicting step of generating a motion-compensated prediction image bydetecting movement amounts in predetermined partial image region units;a smoothing step of performing smoothing of pixels located at theboundaries of adjoining partial image regions on the basis ofpredetermined evaluation criteria, with respect to the prediction imageobtained by the motion compensation predicting step; a predictionresidual decoding step of decoding a prediction residual signal from theencoding side; and an adding step of obtaining a decoded image by addingtogether a decoded prediction residual signal obtained by the predictionresidual decoding step, and the smoothed prediction image.
 41. The imagedecoding method as defined in claim 40, characterized in that the motioncompensation predicting step obtains the prediction image by usingdecoded local decoding image data as a reference image.
 42. The imagedecoding method as defined in claim 40, characterized in that thesmoothing step performs the smoothing in accordance with a differencevalue for the movement amount between the adjoining partial imageregions.
 43. The image decoding method as defined in claim 40,characterized in that the smoothing step performs the smoothing inaccordance with the difference in decoding method between the adjoiningpartial image regions.
 44. The image decoding method as defined in claim40, characterized in that the smoothing step performs the smoothing inaccordance with the difference in image prediction method between theadjoining partial image regions.
 45. The image decoding method asdefined in claim 40, characterized in that the smoothing step performsthe smoothing in accordance with the error margin amount between theprediction image prior to smoothing, and the reference image from whichthe prediction image obtained by the motion compensation predicting stepis generated.
 46. The image decoding method as defined in claim 40,characterized in that the smoothing step comprises: an activity levelsetting step of determining the activity level between partial imageregions which is determined on the basis of at least one of: adifference value for the movement amount between the adjoining partialimage regions, a difference in coding method between the adjoiningpartial image regions, a difference in image prediction method betweenthe adjoining partial image regions, and an error margin amount betweenthe prediction image prior to smoothing, and the reference image fromwhich the prediction image obtained by the motion compensationpredicting means is generated; an adaptive smoothing step ofestablishing the intensity of the smoothing on the basis of the activitylevel and then performing smoothing of pixels located at the boundariesbetween the partial image regions; and a post-processing step ofperforming suppression processing, of the results of smoothing, by meansof a threshold value established in accordance with the activity level.47. The image decoding method as defined in claim 46, characterized inthat the activity level setting step sets the activity level forindividual partial image regions and input to the adaptive smoothingstep the larger value of values for the activity level obtained bycomparing partial image regions which are subjected to smoothing. 48.The image decoding method as defined in claim 46, characterized in thatthe activity level setting step sets the activity level for boundariesbetween the partial image regions and input the activity level to theadaptive smoothing step.
 49. The image decoding method as defined inclaim 46, characterized in that the adaptive smoothing step changes thenumber of pixels subjected to smoothing in accordance with the activitylevel obtained by the activity level setting step.
 50. The imagedecoding method as defined in claim 46, characterized in that theadaptive smoothing step switches filter characteristics for performingthe smoothing in accordance with the activity level obtained by theactivity level setting step.
 51. The image decoding method as defined inclaim 50, characterized in that the adaptive smoothing step switches thefilter characteristics on the basis of a bit that indicates the type offilter characteristics for performing the smoothing, this bit beingdecoded from compressed input data from the encoding side.
 52. The imagedecoding method as defined in claim 40, characterized in that thesmoothing processing is controlled on the basis of a bit that indicatesthe presence or absence of the smoothing, this bit being decoded fromthe compressed input data.